Backup Navigation System for Unmanned Aerial Vehicles

ABSTRACT

Described is a method that involves operating an unmanned aerial vehicle (UAV) to begin a flight, where the UAV relies on a navigation system to navigate to a destination. During the flight, the method involves operating a camera to capture images of the UAV&#39;s environment, and analyzing the images to detect features in the environment. The method also involves establishing a correlation between features detected in different images, and using location information from the navigation system to localize a feature detected in different images. Further, the method involves generating a flight log that includes the localized feature. Also, the method involves detecting a failure involving the navigation system, and responsively operating the camera to capture a post-failure image. The method also involves identifying one or more features in the post-failure image, and determining a location of the UAV based on a relationship between an identified feature and a localized feature.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of co-owned U.S. patent applicationSer. No. 15/703,948, filed on Sep. 13, 2017, which is incorporatedherein by reference in its entirety and for all purposes.

BACKGROUND

An unmanned aerial vehicle (“UAV”), which may also be referred to as anautonomous vehicle, is a vehicle capable of travel without aphysically-present human operator. A UAV may operate in a remote-controlmode, in an autonomous mode, or in a partially autonomous mode.

When a UAV operates in a remote-control mode, a pilot or driver that isat a remote location can control the UAV via commands that are sent tothe UAV via a wireless link. When the UAV operates in autonomous mode,the UAV typically moves based on pre-programmed navigation waypoints,dynamic automation systems, or a combination of these. Further, someUAVs can operate in both a remote-control mode and an autonomous mode,and in some instances may do so simultaneously. For instance, a remotepilot or driver may wish to leave navigation to an autonomous systemwhile manually performing another task, such as operating a mechanicalsystem for picking up objects, as an example.

Various types of UAVs exist for various different environments. Forinstance, UAVs exist for operation in the air. Examples includequad-copters and tail-sitter UAVs, among others. UAVs also exist forhybrid operations in which multi-environment operation is possible.

SUMMARY

In practice, unmanned aerial vehicles (UAVs) include a primarynavigation system that can help navigate the UAV to a desireddestination. For various reasons, however, the primary navigation systemcan encounter a failure event that can render the primary navigationsystem inoperable. If a failure event occurs during the flight,undesirable results can take place, such as the UAV crashing.Accordingly, it is desirable to equip the UAV with a backup navigationsystem that can replace at least some of the navigational functionalityof the primary navigation system in the case of a failure event, or thatcan be used to cross-check the integrity of the primary navigationsystem to protect it against malicious attacks or hazardous andmisleading information (HMI). Disclosed herein is a backup navigationsystem that can generate and/or update a backup map of the UAV's flightpath to a destination based on environmental imagery captured by the UAVas the flight progresses, such that this map can be utilized to providecontinued navigation to the UAV so it can safely complete its mission orexecute a landing at a safe landing zone (e.g., home base) in the eventthat the primary navigation system fails.

In one aspect, a method is provided. The method can involve operating anunmanned aerial vehicle (UAV) to begin a flight along a first flightpath to a destination, where the UAV uses a primary navigation system tonavigate to the destination. The method can also involve operating animage capture device coupled to the UAV and having a field-of-view (FOV)of an environment of the UAV to capture a first set of images as the UAVtravels along the first flight path. The method can also involveanalyzing the images to localize visual features within the environment,and generating a local flight log for the first flight path, whichcomprises location information and image data for the detected features.Further, the method can involve detecting a failure event involving theprimary navigation system and responsive to detecting the failure event,capturing at least one post-failure image using the image capturedevice. Yet further the method can involve identifying, in the at leastone post-failure image, one or more visual features within theenvironment, and determining a current location of the UAV bydetermining a current location of the UAV by determining a relationshipbetween a first identified visual feature and a first localized visualfeature.

In another aspect, an unmanned aerial vehicle is provided. The unmannedaerial vehicle (UAV) can include a primary navigation system thatprovides the UAV with navigational functionality and a backup navigationsystem configured to replace the navigational functionality of theprimary navigation system upon detecting a failure event involving theprimary navigation system. In particular, the backup navigation systemcan include an image capture device configured to capture images of anenvironment of the UAV. Further, the UAV can include a controller thatcan be configured to operate the UAV to begin a flight along a firstflight path to a destination, and operate the backup navigation systemin a feature localization mode in which the controller: (i) causes theimage capture device to capture a first set of images along the firstflight path, and (ii) analyzes the images to localize a set of visualfeatures within the environment, where each localized visual feature isassociated with a respective geolocation, and where the set of localizedfeatures is stored in a memory location. Yet further, the controller canbe configured to detect a failure event involving the primary navigationsystem, and responsive to detecting the failure event, operate thebackup navigation system in a UAV localization mode in which thecontroller: (i) causes the image capture device to capture at least oneimage of the environment, (ii) identifies in the at least one imagevisual features within the environment, and (iii) compares theidentified visual features to the set localized visual features storedin the memory location.

In yet another aspect, a method is provided. The method can includeoperating an unmanned aerial vehicle (UAV) to begin a flight to adestination, wherein the UAV uses a primary navigation system tonavigate to the destination. The method can also include operating animage capture device coupled to the UAV and having a field-of-view (FOV)of an environment of the UAV to capture a first set of images during theUAV's flight, and analyzing the images to identify visual featureswithin the environment. Also, the method can include determining acurrent location of the UAV by determining a relationship between one ofthe identified visual features and a localized visual feature in a mapstored in a memory of the UAV. Further, the method can include comparingthe current location to location data from the primary navigationsystem, and responsive to detecting a discrepancy between the currentlocation and the location data from the primary navigation system,outputting an alert indicating failure of the primary navigation system.

These as well as other aspects, advantages, and alternatives will becomeapparent to those of ordinary skill in the art by reading the followingdetailed description with reference where appropriate to theaccompanying drawings. Further, it should be understood that thedescription provided in this summary section and elsewhere in thisdocument is intended to illustrate the claimed subject matter by way ofexample and not by way of limitation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a simplified illustration of an unmanned aerial vehicle,according to an example implementation.

FIG. 1B is a simplified illustration of an unmanned aerial vehicle,according to an example implementation.

FIG. 1C is a simplified illustration of an unmanned aerial vehicle,according to an example implementation.

FIG. 1D is a simplified illustration of an unmanned aerial vehicle,according to an example implementation.

FIG. 1E is a simplified illustration of an unmanned aerial vehicle,according to an example implementation.

FIG. 2 is a simplified block diagram illustrating a UAS deploymentsystem, according to an example implementation.

FIG. 3 is a simplified illustration of an unmanned aerial vehicleequipped with a backup navigation system, according to an exampleimplementation.

FIG. 4A is a side view of an unmanned aerial vehicle flying to adestination, according to an example implementation.

FIG. 4B is an image captured by an unmanned aerial vehicle, according toan example implementation.

FIG. 4C is another side view of an unmanned aerial vehicle flying to adestination, according to an example implementation.

FIG. 4D is another image captured by an unmanned aerial vehicle,according to an example implementation.

FIG. 4E is another side view of an unmanned aerial vehicle flying to adestination, according to an example implementation.

FIG. 4F is another image captured by an unmanned aerial vehicle,according to an example implementation.

FIG. 5 is an image of a terrain captured by an unmanned aerial vehicle,according to an example implementation.

FIG. 6 is a simplified block diagram illustrating components of anunmanned aerial vehicle, according to an example implementation.

FIG. 7 illustrates a schematic diagram of a server, according to anexample implementation.

FIG. 8 illustrates an example flowchart, according to an exampleimplementation.

FIG. 9 illustrates another example flowchart, according to an exampleimplementation.

FIG. 10 illustrates another example flowchart, according to an exampleimplementation.

DETAILED DESCRIPTION

Exemplary methods and systems are described herein. It should beunderstood that the word “exemplary” is used herein to mean “serving asan example, instance, or illustration.” Any implementation or featuredescribed herein as “exemplary” or “illustrative” is not necessarily tobe construed as preferred or advantageous over other implementations orfeatures. In the figures, similar symbols typically identify similarcomponents, unless context dictates otherwise. The exampleimplementations described herein are not meant to be limiting. It willbe readily understood that the aspects of the present disclosure, asgenerally described herein, and illustrated in the figures, can bearranged, substituted, combined, separated, and designed in a widevariety of different configurations, all of which are contemplatedherein.

I. Overview

An Unmanned Aerial Vehicle (UAV) can perform a task that involves flyingto one or more destinations, e.g., picking up and/or delivering an item.In practice, the UAV can include a navigation system that the UAV relieson to navigate to a particular location or locations. This navigationsystem may be referred to herein as the UAV's “primary navigationsystem.” In example embodiments, the UAV's primary navigation system canrely upon a signal from a on the Global Positioning System (GPS) toprovide coordinates for navigation.

However, the primary navigation system could fail or be renderedtemporarily inoperable in a manner that affects the ability of the UAVto remain on its flight path and/or safely navigate to a targetlocation. In some instances, the failure event could render the systeminoperable, which could result in a catastrophic event, such as the UAVcrashing. In other instances, GPS can be spoofed, which can result inproblems such as an unauthorized party taking control of it. Therefore,it is desirable to equip the UAV with a backup navigation system thatcan replace at least some of the navigational functionality of theprimary navigation system in the event that the primary navigationsystem becomes inoperable or is spoofed (e.g., an unauthorized partycontrolling GPS).

Current backup navigation systems typically employ one of two classes ofcomputer vision algorithms. The first class includes Visual Odometry(VO) or Visual-Inertial Odometry (VIO) algorithms that are capable ofproviding relative position estimates (i.e., an estimate of thedifference in position of the UAV between two points in time). Thesecond class includes Simultaneous Localization And Mapping (SLAM)algorithms that are capable of providing global position estimates(i.e., an estimate of the position of the UAV at all times with respectto a some fixed reference point, such as the starting location of theUAV). However, both of these classes have shortcomings. Specifically, aVIO dependent system only provides relative position estimates, which,on its own, is not typically sufficient for autonomous navigation overlong distances due to increasing position error. Further, the complexityof SLAM algorithms means that the SLAM dependent system is typicallyunable to provide real-time location data, which is required (or atleast highly desirable) for a reliable autonomous navigation system.Accordingly, example embodiments can help to provide a more reliablebackup navigation system, which utilizes real-time image data fornavigation.

More specifically, example backup navigation systems can providereal-time updates to the UAV so that UAV can navigate to a safe landingzone in the event that the primary navigation system fails. In the eventof a failure, such a backup navigation system can navigate the UAV usingmaps that were generated during flight, using actual imagery captured bythe UAV, before the failure occurred. In particular, during the UAV'sflight to a destination, before the failure event occurs, the backupnavigation system can operate in a first mode in which the system canacquire and use environmental image data to generate and update a map ofan environment along the UAV's flight path. Then, when a failure eventoccurs, the backup navigation system can transition to operating in asecond mode in which the system can use the generated maps to navigatethe UAV back to a safe landing zone.

More specifically, the backup navigation system can begin to operate inthe first mode, also called a “feature localization mode,” when the UAVbegins its flight to the destination. Note that when the backupnavigation system is operating in the feature localization mode, the UAVis relying on the primary navigation system for navigation. Furthermore,in the feature localization mode, the backup navigation system cangenerate and/or update a map of the UAV's environment during the UAV'sflight to the destination. In an implementation, the system can generateand/or update the map by detecting features of the environment near theUAV's flight path to the destination. In an example, the system candetect features of the environment by capturing images of theenvironment and processing the images to detect features of theenvironment that were captured in the images. To facilitate capturingthe images, the system can include an image capture device that isconfigured to capture images of the environment during the UAV's flight.

The backup navigation system can then determine a respective locationfor one or more of the detected features (i.e., localize the detectedfeatures). In an example, the system can localize a feature by using aknown position and orientation of the UAV (provided by the primarynavigation system) and multiple observations of the feature totriangulate the location of the feature. The localized features can beused to generate a flight log or map of the UAV's flight path to thedestination. That is, each detected feature along the flight path isassociated with at least a location in the environment. In someexamples, a feature that was detected in a particular image can beassociated with one or more characteristics indicative of the conditionsin which the particular image was captured, such as time of day theimage was captured, date the image was captured, weather at the locationthat the image was captured, among other characteristics.

In an embodiment, the backup navigation system can operate in thefeature localization mode until the system detects that the primarynavigation system is inoperable, at which point the backup navigationsystem can responsively transition to operating in the second mode, alsocalled a “UAV localization mode.” In the UAV localization mode, thebackup navigation system can be responsible for navigating the UAV to asafe landing zone, such as the home base of the UAV. In animplementation, responsive to the primary navigation system failing, thebackup navigation system can navigate the UAV to its home base by flyingthe UAV along substantially the same flight path that the UAV flew toreach its current position (i.e., the position at which the primarynavigation system failed). To navigate along the return flight path, thebackup navigation system can use the map or flight log that wasgenerated when the system was operating in the feature localizationmode.

More specifically, the backup navigation system can use the map bycapturing images of the UAV's current environment and processing theimages to extract features of the environment. Then, the system cancompare the extracted features to the localized features that are partof the generated map of the flight path. If the backup navigation systemdetects a relationship between the extracted features and the localizedfeatures, the backup navigation system can use the relationship and themap (i.e., known locations of the localized features) to determine acurrent location of the UAV. Based on the current location of the UAV,the backup navigation system can determine at least a portion of aflight path to the safe zone. For example, the backup navigation systemcan determine a portion of a flight path to the home base, where theflight path to the home base is substantially the same flight path asthe initial flight path that the UAV flew to the UAV's current position.By determining a return flight path that is substantially along theinitial flight path, the backup navigation system can use the generatedmap to navigate since the map includes localized features of the initialflight path. Accordingly, as the UAV travels along the return the flightpath, the backup navigation system can use the map to identify localizedfeatures, and based on the localized features, can determine a currentposition of the UAV.

In some instances, the backup navigation system can determine tocontinue travelling to the original destination (i.e., continue themission). In other instances, for various reasons (e.g., fuelconsiderations), the backup navigation system can determine to fly theUAV to a safe zone that is not in an area that was traversed by the UAV.In either instance, since the UAV did not traverse the area, the UAV mayhave not generated a map of the area, and therefore the backupnavigation system may not be able to navigate to the desired safe zone.To overcome this obstacle, also disclosed herein is a method ofcompiling a database of localized features from images captured by aplurality of UAVs that are operating in a given area over a given amountof time. In particular, a controller of the database can process theimages (and the features extracted from thereof) to generate one or moremaps of the given area. Then any UAV whose flight path overlaps thegiven area will receive a copy of the one or maps so that the UAV'sbackup navigation system can use the maps to navigate to a safe zone inthe given area.

Within examples, the controller of the database can generate spatial andtemporal maps of a given area. That is, the controller can generate morethan one map of the given area, where each map is associated with one ormore temporal characteristics, such as a time of day and date. The timeand date may be the time and date at which the images that were used togenerate a map were captured. Then a UAV can use a spatial and atemporal map that is associated with the time and date of the UAV'sflight. By using the spatial and temporal map that is associated withthe time and date of the UAV's flight, the backup navigation system ismore likely to detect known features in the environment.

Such a backup navigation system also provides improvement overimage-based navigation systems since the backup navigation system canaccount for temporal changes in the environment. Generally, the visualfeatures of an environment can vary depending on the time of the day,the date, weather conditions, season, etc. By generating a localized mapwhen flying to a destination, the backup navigation system can capturevisual features in the environment in their most recent state. Thus, thebackup system has a higher probability of identifying the visualfeatures when operating in the UAV localization mode. As also explainedherein, the backup navigation system can associate the maps withtemporal characteristics such as date or season.

In addition to providing backup navigation, the backup navigation systemdisclosed herein can also perform other functions, such as checking theintegrity or accuracy of the navigation provided by the primarynavigation system. In particular, the integrity of the primarynavigation system can be compromised due to undetected malfunctions ormalicious third-party attempts to compromise the system (e.g. GPSspoofing). In either case, the compromised primary navigation system cangive hazardous and misleading information (HMI) to the flight controlsystem, leading to potential safety issues. In an implementation of thisfunction, the backup navigation system can periodically determine theUAV's location (e.g., using the methods described herein), and can thencompare the UAV location to the location determined by the primarynavigation system. In one example, if there is a discrepancy between thetwo locations that is a greater than a threshold, the backup navigationsystem can determine that the integrity of the primary navigation systemhas been compromised. In such scenarios, the backup navigation systemcan also override the primary navigation system to provide the UAV withan accurate navigation solution.

II. Illustrative UAVs

Herein, the terms “unmanned aircraft vehicle” and “UAV” refer to anyautonomous or semi-autonomous vehicle that is capable of performing somefunctions without a physically present human pilot.

A UAV can take various forms. For example, a UAV may take the form of afixed-wing aircraft, a glider aircraft, a tail-sitter aircraft, a jetaircraft, a ducted fan aircraft, a lighter-than-air dirigible such as ablimp or steerable balloon, a rotorcraft such as a helicopter ormulticopter, and/or an ornithopter, among other possibilities. Further,the terms “drone,” “unmanned aerial vehicle system” (UAVs), or “unmannedaerial system” (UAS) may also be used to refer to a UAV.

FIG. 1A is a simplified illustration providing various views of a UAV,according to an example implementation. In particular, FIG. 1A shows anexample of a fixed-wing UAV 1100 a, which may also be referred to as anairplane, an aeroplane, a biplane, a glider, or a plane, among otherpossibilities. The fixed-wing UAV 1100 a, as the name implies, hasstationary wings 1102 that generate lift based on the wing shape and thevehicle's forward airspeed. For instance, the two wings 1102 may have anairfoil-shaped cross section to produce an aerodynamic force on the UAV1100 a.

As depicted, the fixed-wing UAV 1100 a may include a wing body orfuselage 1104.

The wing body 1104 may contain, for example, control electronics such asan inertial measurement unit (IMU) and/or an electronic speedcontroller, batteries, other sensors, and/or a payload, among otherpossibilities. The illustrative UAV 1100 a may also include landing gear(not shown) to assist with controlled take-offs and landings. In otherimplementations, other types of UAVs without landing gear are alsopossible.

The UAV 1100 a further includes propulsion units 1106 positioned on thewings 1106 (or fuselage), which can each include a motor, shaft, andpropeller, for propelling the UAV 1100 a. Stabilizers 1108 (or fins) mayalso be attached to the UAV 1110 a to stabilize the UAV's yaw (turn noseleft or right) during flight. In some implementations, the UAV 1100 amay be also be configured to function as a glider. To do so, UAV 1100 amay power off its motor, propulsion units, etc., and glide for a periodof time. In the UAV 1100 a, a pair of rotor supports 1110 extend beneaththe wings 1106, and a plurality of rotors 1112 are attached rotorsupports 1110. Rotors 1110 may be used during a hover mode wherein theUAV 1110 a is descending to a delivery location, or ascending followinga delivery. In the example UAV 1100 a, stabilizers 1108 are shownattached to the rotor supports 1110.

During flight, the UAV 1100 a may control the direction and/or speed ofits movement by controlling its pitch, roll, yaw, and/or altitude. Forexample, the stabilizers 1108 may include one or more rudders 1108 a forcontrolling the UAV's yaw, and the wings 1102 may include one or moreelevators for controlling the UAV's pitch and/or one or more ailerons1102 a for controlling the UAV's roll. As another example, increasing ordecreasing the speed of all the propellers simultaneously can result inthe UAV 1100 a increasing or decreasing its altitude, respectively.

Similarly, FIG. 1B shows another example of a fixed-wing UAV 120. Thefixed-wing UAV 120 includes a fuselage 122, two wings 124 with anairfoil-shaped cross section to provide lift for the UAV 120, a verticalstabilizer 126 (or fin) to stabilize the plane's yaw (turn left orright), a horizontal stabilizer 128 (also referred to as an elevator ortailplane) to stabilize pitch (tilt up or down), landing gear 130, and apropulsion unit 132, which can include a motor, shaft, and propeller.

FIG. 1C shows an example of a UAV 140 with a propeller in a pusherconfiguration. The term “pusher” refers to the fact that a propulsionunit 142 is mounted at the back of the UAV and “pushes” the vehicleforward, in contrast to the propulsion unit being mounted at the frontof the UAV. Similar to the description provided for FIGS. 1A and 1B,FIG. 1C depicts common structures used in a pusher plane, including afuselage 144, two wings 146, vertical stabilizers 148, and thepropulsion unit 142, which can include a motor, shaft, and propeller.

FIG. 1D shows an example of a tail-sitter UAV 160. In the illustratedexample, the tail-sitter UAV 160 has fixed wings 162 to provide lift andallow the UAV 160 to glide horizontally (e.g., along the x-axis, in aposition that is approximately perpendicular to the position shown inFIG. 1D). However, the fixed wings 162 also allow the tail-sitter UAV160 to take off and land vertically on its own.

For example, at a launch site, the tail-sitter UAV 160 may be positionedvertically (as shown) with its fins 164 and/or wings 162 resting on theground and stabilizing the UAV 160 in the vertical position. Thetail-sitter UAV 160 may then take off by operating its propellers 166 togenerate an upward thrust (e.g., a thrust that is generally along they-axis). Once at a suitable altitude, the tail-sitter UAV 160 may useits flaps 168 to reorient itself in a horizontal position, such that itsfuselage 170 is closer to being aligned with the x-axis than the y-axis.Positioned horizontally, the propellers 166 may provide forward thrustso that the tail-sitter UAV 160 can fly in a similar manner as a typicalairplane.

Many variations on the illustrated fixed-wing UAVs are possible. Forinstance, fixed-wing UAVs may include more or fewer propellers, and/ormay utilize a ducted fan or multiple ducted fans for propulsion.Further, UAVs with more wings (e.g., an “x-wing” configuration with fourwings), with fewer wings, or even with no wings, are also possible.

As noted above, some implementations may involve other types of UAVs, inaddition to or in the alternative to fixed-wing UAVs. For instance, FIG.1E shows an example of a rotorcraft that is commonly referred to as amulticopter 180. The multicopter 180 may also be referred to as aquadcopter, as it includes four rotors 182. It should be understood thatexample implementations may involve a rotorcraft with more or fewerrotors than the multicopter 180. For example, a helicopter typically hastwo rotors. Other examples with three or more rotors are possible aswell. Herein, the term “multicopter” refers to any rotorcraft havingmore than two rotors, and the term “helicopter” refers to rotorcrafthaving two rotors.

Referring to the multicopter 180 in greater detail, the four rotors 182provide propulsion and maneuverability for the multicopter 180. Morespecifically, each rotor 182 includes blades that are attached to amotor 184. Configured as such, the rotors 182 may allow the multicopter180 to take off and land vertically, to maneuver in any direction,and/or to hover. Further, the pitch of the blades may be adjusted as agroup and/or differentially, and may allow the multicopter 180 tocontrol its pitch, roll, yaw, and/or altitude.

It should be understood that references herein to an “unmanned” aerialvehicle or UAV can apply equally to autonomous and semi-autonomousaerial vehicles. In an autonomous implementation, all functionality ofthe aerial vehicle is automated; e.g., pre-programmed or controlled viareal-time computer functionality that responds to input from varioussensors and/or pre-determined information. In a semi-autonomousimplementation, some functions of an aerial vehicle may be controlled bya human operator, while other functions are carried out autonomously.Further, in some implementations, a UAV may be configured to allow aremote operator to take over functions that can otherwise be controlledautonomously by the UAV. Yet further, a given type of function may becontrolled remotely at one level of abstraction and performedautonomously at another level of abstraction. For example, a remoteoperator could control high level navigation decisions for a UAV, suchas by specifying that the UAV should travel from one location to another(e.g., from a warehouse in a suburban area to a delivery address in anearby city), while the UAV's navigation system autonomously controlsmore fine-grained navigation decisions, such as the specific route totake between the two locations, specific flight controls to achieve theroute and avoid obstacles while navigating the route, and so on.

More generally, it should be understood that the example UAVs describedherein are not intended to be limiting. Example implementations mayrelate to, be implemented within, or take the form of any type ofunmanned aerial vehicle.

III. Illustrative UAV Deployment Systems

The backup navigation system (and the corresponding methods) disclosedherein can relate to, be implemented within, or any kind of UAV or UAVsystem, such as UAVs that are configured for environment monitoring,photography, surveying, among other examples systems. Described below isan illustrative UAV deployment system. However, it should be understoodthat this system is not intended to be limiting, and that other systemsare also possible.

UAV deployment systems may be implemented in order to provide variousUAV-related services. In particular, UAVs may be provided at a number ofdifferent launch sites that may be in communication with regional and/orcentral control systems. Such a distributed UAV deployment system mayallow UAVs to be quickly deployed to provide services across a largegeographic area (e.g., that is much larger than the flight range of anysingle UAV). For example, UAVs capable of carrying payloads may bedistributed at a number of launch sites across a large geographic area(possibly even throughout an entire country, or even worldwide), inorder to provide on-demand transport of various items to locationsthroughout the geographic area. FIG. 2 is a simplified block diagramillustrating a distributed UAV deployment system 200, according to anexample implementation.

In the illustrative UAV deployment system 200, an access system 202 mayallow for interaction with, control of, and/or utilization of a networkof UAVs 204. In some implementations, an access system 202 may be acomputing device that allows for human-controlled dispatch of UAVs 204.As such, the control system may include or otherwise provide a userinterface through which a user can access and/or control the UAVs 204.

In some implementations, dispatch of the UAVs 204 may additionally oralternatively be accomplished via one or more automated processes. Forinstance, the access system 202 may dispatch one of the UAVs 204 totransport a payload to a target location, and the UAV may autonomouslynavigate to the target location by utilizing various on-board sensors,such as a GPS receiver and/or other various navigational sensors.

Further, the access system 202 may provide for remote operation of aUAV. For instance, the access system 202 may allow an operator tocontrol the flight of a UAV via its user interface. As a specificexample, an operator may use the access system 202 to dispatch a UAV 204to a target location. The UAV 204 may then autonomously navigate to thegeneral area of the target location. At this point, the operator may usethe access system 202 to take control of the UAV 204 and navigate theUAV to the target location (e.g., to a particular person to whom apayload is being transported). Other examples of remote operation of aUAV are also possible.

In an illustrative implementation, the UAVs 204 may take various forms.For example, each of the UAVs 204 may be a UAV such as those illustratedin FIGS. 1A-1E. However, UAV deployment system 200 may also utilizeother types of UAVs without departing from the scope of the invention.In some implementations, all of the UAVs 204 may be of the same or asimilar configuration. However, in other implementations, the UAVs 204may include a number of different types of UAVs. For instance, the UAVs204 may include a number of types of UAVs, with each type of UAV beingconfigured for a different type or types of payload deliverycapabilities.

The UAV deployment system 200 may further include a remote device 206,which may take various forms. Generally, the remote device 206 may beany device through which a direct or indirect request to dispatch a UAVcan be made. (Note that an indirect request may involve anycommunication that may be responded to by dispatching a UAV, such asrequesting a package delivery). In an example implementation, the remotedevice 206 may be a mobile phone, tablet computer, laptop computer,personal computer, or any network-connected computing device. Further,in some instances, the remote device 206 may not be a computing device.As an example, a standard telephone, which allows for communication viaplain old telephone service (POTS), may serve as the remote device 206.Other types of remote devices are also possible.

Further, the remote device 206 may be configured to communicate withaccess system 202 via one or more types of communication network(s) 208.For example, the remote device 206 may communicate with the accesssystem 202 (or a human operator of the access system 202) bycommunicating over a POTS network, a cellular network, and/or a datanetwork such as the Internet. Other types of networks may also beutilized.

In some implementations, the remote device 206 may be configured toallow a user to request delivery of one or more items to a desiredlocation. For example, a user could request UAV delivery of a package totheir home via their mobile phone, tablet, or laptop. As anotherexample, a user could request dynamic delivery to wherever they arelocated at the time of delivery. To provide such dynamic delivery, theUAV deployment system 200 may receive location information (e.g., GPScoordinates, etc.) from the user's mobile phone, or any other device onthe user's person, such that a UAV can navigate to the user's location(as indicated by their mobile phone).

In an illustrative arrangement, the central dispatch system 210 may be aserver or group of servers, which is configured to receive dispatchmessages requests and/or dispatch instructions from the access system202. Such dispatch messages may request or instruct the central dispatchsystem 210 to coordinate the deployment of UAVs to various targetlocations. The central dispatch system 210 may be further configured toroute such requests or instructions to one or more local dispatchsystems 212. To provide such functionality, the central dispatch system210 may communicate with the access system 202 via a data network, suchas the Internet or a private network that is established forcommunications between access systems and automated dispatch systems.

In the illustrated configuration, the central dispatch system 210 may beconfigured to coordinate the dispatch of UAVs 204 from a number ofdifferent local dispatch systems 212. As such, the central dispatchsystem 210 may keep track of which UAVs 204 are located at which localdispatch systems 212, which UAVs 204 are currently available fordeployment, and/or which services or operations each of the UAVs 204 isconfigured for (in the event that a UAV fleet includes multiple types ofUAVs configured for different services and/or operations). Additionallyor alternatively, each local dispatch system 212 may be configured totrack which of its associated UAVs 204 are currently available fordeployment and/or are currently in the midst of item transport.

In some cases, when the central dispatch system 210 receives a requestfor UAV-related service (e.g., transport of an item) from the accesssystem 202, the central dispatch system 210 may select a specific UAV204 to dispatch. The central dispatch system 210 may accordinglyinstruct the local dispatch system 212 that is associated with theselected UAV to dispatch the selected UAV. The local dispatch system 212may then operate its associated deployment system 214 to launch theselected UAV. In other cases, the central dispatch system 210 mayforward a request for a UAV-related service to a local dispatch system212 that is near the location where the support is requested and leavethe selection of a particular UAV 204 to the local dispatch system 212.

In an example configuration, the local dispatch system 212 may beimplemented as a computing device at the same location as the deploymentsystem(s) 214 that it controls. For example, the local dispatch system212 may be implemented by a computing device installed at a building,such as a warehouse, where the deployment system(s) 214 and UAV(s) 204that are associated with the particular local dispatch system 212 arealso located. In other implementations, the local dispatch system 212may be implemented at a location that is remote to its associateddeployment system(s) 214 and UAV(s) 204.

Numerous variations on and alternatives to the illustrated configurationof the UAV deployment system 200 are possible. For example, in someimplementations, a user of the remote device 206 could request deliveryof a package directly from the central dispatch system 210. To do so, anapplication may be implemented on the remote device 206 that allows theuser to provide information regarding a requested delivery, and generateand send a data message to request that the UAV deployment system 200provide the delivery. In such an implementation, the central dispatchsystem 210 may include automated functionality to handle requests thatare generated by such an application, evaluate such requests, and, ifappropriate, coordinate with an appropriate local dispatch system 212 todeploy a UAV.

Further, some or all of the functionality that is attributed herein tothe central dispatch system 210, the local dispatch system(s) 212, theaccess system 202, and/or the deployment system(s) 214 may be combinedin a single system, implemented in a more complex system, and/orredistributed among the central dispatch system 210, the local dispatchsystem(s) 212, the access system 202, and/or the deployment system(s)214 in various ways.

Yet further, while each local dispatch system 212 is shown as having twoassociated deployment systems 214, a given local dispatch system 212 mayalternatively have more or fewer associated deployment systems 214.Similarly, while the central dispatch system 210 is shown as being incommunication with two local dispatch systems 212, the central dispatchsystem 210 may alternatively be in communication with more or fewerlocal dispatch systems 212.

In a further aspect, the deployment systems 214 may take various forms.In general, the deployment systems 214 may take the form of or includesystems for physically launching one or more of the UAVs 204. Suchlaunch systems may include features that provide for an automated UAVlaunch and/or features that allow for a human-assisted UAV launch.Further, the deployment systems 214 may each be configured to launch oneparticular UAV 204, or to launch multiple UAVs 204.

The deployment systems 214 may further be configured to provideadditional functions, including for example, diagnostic-relatedfunctions such as verifying system functionality of the UAV, verifyingfunctionality of devices that are housed within a UAV (e.g., a payloaddelivery apparatus), and/or maintaining devices or other items that arehoused in the UAV (e.g., by monitoring a status of a payload such as itstemperature, weight, etc.).

In some implementations, the deployment systems 214 and theircorresponding UAVs 204 (and possibly associated local dispatch systems212) may be strategically distributed throughout an area such as a city.For example, the deployment systems 214 may be strategically distributedsuch that each deployment system 214 is proximate to one or more payloadpickup locations (e.g., near a restaurant, store, or warehouse).However, the deployment systems 214 (and possibly the local dispatchsystems 212) may be distributed in other ways, depending upon theparticular implementation. As an additional example, kiosks that allowusers to transport packages via UAVs may be installed in variouslocations. Such kiosks may include UAV launch systems, and may allow auser to provide their package for loading onto a UAV and pay for UAVshipping services, among other possibilities. Other examples are alsopossible.

In a further aspect, the UAV deployment system 200 may include or haveaccess to a user-account database 216. The user-account database 216 mayinclude data for a number of user accounts, and which are eachassociated with one or more person. For a given user account, theuser-account database 216 may include data related to or useful inproviding UAV-related services. Typically, the user data associated witheach user account is optionally provided by an associated user and/or iscollected with the associated user's permission.

Further, in some implementations, a person may be required to registerfor a user account with the UAV deployment system 200, if they wish tobe provided with UAV-related services by the UAVs 204 from UAVdeployment system 200. As such, the user-account database 216 mayinclude authorization information for a given user account (e.g., ausername and password), and/or other information that may be used toauthorize access to a user account.

In some implementations, a person may associate one or more of theirdevices with their user account, such that they can access the servicesof UAV deployment system 200. For example, when a person uses anassociated mobile phone, e.g., to place a call to an operator of theaccess system 202 or send a message requesting a UAV-related service toa dispatch system, the phone may be identified via a unique deviceidentification number, and the call or message may then be attributed tothe associated user account. Other examples are also possible.

IV. Navigation System

In line with the discussion above, a UAV can include a primarynavigation system that can navigate the UAV to a destination, which canbe specified in various formats, such as coordinates. The primarynavigation system can use the destination information (e.g., GPScoordinates) to determine how to navigate to the destination. Forinstance, the primary navigation system can generate a flight path froma starting position to the destination, and can then navigate the UAValong the flight path.

However, as explained above, the primary navigation system can encountera failure event during the UAV's flight. The failure event can causesignificant failures or losses, such as the UAV not reaching itsdestination, or even worse, a crash. To help avoid such losses orfailures, it is desirable to equip the UAV with a backup navigationsystem that can replace at least some of the navigational functionalityof the primary navigation system in the case that the primary navigationsystem encounters a failure event.

Accordingly, disclosed herein is a backup navigation system that canprovide the UAV with navigational functionality in the event that theprimary navigation system fails. Within examples, the backup navigationsystem can operate using resources of the UAV and can operate withoutrelying on a remote server. Thus, even if the failure event affects allof the communication systems of the UAV, the backup navigation systemcan provide the UAV with navigational functionality. For instance, thebackup navigation system can navigate the UAV to a safe landing area inresponse to detecting that the UAV and/or the primary navigation systemhas encountered a failure event. In an implementation of the backupnavigation system, the system can provide navigational functionality bygenerating and/or updating a map of the UAV's flight path during theUAV's flight to a destination. Then, if a failure event occurs, thebackup navigation system can use the generated map to navigate the UAVto a safe landing area.

In an embodiment, the backup navigation system can generate and/orupdate a map of the UAV's flight path by capturing images of the UAV'senvironment during the UAV's flight along the flight path. As describedbelow, the backup navigation system can generate and/or update the mapof the UAV's flight path by using the captured images and locationinformation associated with the images (which can be received from theprimary navigation system).

FIG. 3 illustrates a UAV 300 equipped with a backup navigation system,according to an exemplary embodiment. As illustrated in FIG. 3, thebackup navigation system can include an image capture device 304 that isconfigured to capture images of the UAV's environment. In some examples,the image capture device 304 can be directly coupled to the body of theUAV 300. For instance, the image capture device 304 can be rotatablycoupled to the body of the UAV 300 so that the backup navigation systemcan control the orientation of the image capture device 304. Further,the image capture device can be directly coupled within or directly tothe body of the UAV 300 or can be coupled to the UAV 300 via a mount orother coupling mechanism. For instance, as illustrated in FIG. 3, theimage capture device 304 is rotatably coupled to a mount 306 coupled toa bottom surface 308 of the UAV 300.

Within examples, the backup navigation system can orient the backupnavigation system in various orientations. As illustrated in FIG. 3, theimage capture device 304 can be oriented perpendicular to the UAV 300.In this arrangement, the image capture device 304 can capture images ofan environment the UAV 300 that is below the UAV 300. The backupnavigation system can also orient the image capture device 304 in otherorientations, such as a forward facing orientation. In the forwardfacing orientation, the image capture device 304 can capture at least aportion of the environment in front of the UAV 300.

The image capture device 304 can be a device that is configured tocapture still images, video, and/or other sensor data. For instance, theimage capture device 304 can be a camera (e.g., a CCD image sensor), aLIDAR device, a time-of-flight camera, a structured light scanner,and/or a stereo camera, among other possibilities. The image capturedevice 304 can be configured to capture images at various resolutions orat different frame rates. In some examples, the specifications (e.g.,resolution) of the image capture device 304 can depend on thespecifications of the UAV 300 (e.g., maximum altitude of the UAV). Forinstance, a UAV that is configured to fly at a high altitude can includean image capture device with a higher resolution than an image capturedevice of a UAV that is configured to fly at a low altitude.

Furthermore, in the UAV configuration illustrated in FIG. 3, the imagecapture device 304 is positioned on the surface 308 of the UAV 300;however, the image capture device 304 can be positioned on other partsof the UAV 300. Further, although FIG. 3 illustrates one image capturedevice 304, more image capture devices can be used, and each can beconfigured to capture the same view, or to capture different views. Forexample, another image capture device can be positioned on a frontsurface of the UAV 300 to capture at least a portion of a view in frontof the UAV 300.

In an embodiment, the UAV 300 can be tasked with flying to a particulardestination. In order for the UAV 300 to fly to the destination, theprimary navigation system of the UAV 300 can determine a flight pathfrom a starting point (e.g., UAV 300's home base) to the destination,where the flight path is a defined path of position and altitude alongwhich the UAV can fly. Within examples, the primary navigation systemcan determine the flight path based on one or more factors, such astime, distance, environmental considerations (e.g., weather), etc.Additionally and/or alternatively, the primary navigation system candetermine a flight path that optimizes for one or more of the factors,which, in some examples, the system can optimize for based on a userpreference. For instance, the primary navigation system can generate oneor more flight paths that reach the destination, and can then select theflight path that can be traversed the quickest.

In an embodiment, the primary navigation system can navigate the UAV 300along the flight path to the destination. In some examples, the primarynavigation system can rely on a satellite or other remote computingdevice for positional information of the UAV 300, which the system canthen use to determine a flight path of the UAV 300. By way of example,the primary navigation system can rely on GPS that can provide thesystem with the UAV's GPS coordinates. The system can then use the UAV'sGPS coordinates to determine the UAV's location with respect to theflight path, and can make adjustments to the flight path and/or theUAV's operation (e.g., speed, flight altitude, etc.) as necessary.

During the UAV 300's flight along the flight path, the primarynavigation system can navigate the UAV 300 as long as the UAV 300 hasnot encountered a failure event. As such, when the primary navigationsystem is functioning, the backup navigation system does not need toprovide the UAV 300 with navigational updates. Nonetheless, the backupnavigation system can operate when the primary navigation system isfunctioning and navigating the UAV 300. In particular, the backupnavigation system can operate in a mode in which the system can generateand/or update maps of the UAV 300's flight path. In this mode (alsoreferred to herein as a “feature localization mode”), the backupnavigation system can detect unique visual features of the UAV 300'senvironment along the UAV 300's flight path, and can determinerespective locations of the unique visual features.

A unique visual feature in the environment is any feature that has oneor more unique characteristics that allow the feature to be detected,such as design characteristics of a structure, shape and dimensions of afeature, aesthetic characteristics (e.g., color), among other examplecharacteristics. Note that detecting a feature does not requireidentifying a type of the feature. Rather detecting a feature caninvolve associating the feature characteristics with a unique identifiersuch that if the feature characteristics are detected again, the UAV candetermine the feature with which the characteristics are associated.Example visual features include a structure (e.g., a building, an oilrig, etc.), an open area (e.g., a park, a parking lot, etc.), a vehicle,a sign (e.g., a billboard, a road sign, etc.), an infrastructurestructure (e.g., a traffic light, a water tower, etc.), an environmentalfeature (e.g., a hill, a flat terrain, a water body, etc.), among otherexamples.

In an embodiment, the backup navigation system can detect unique visualfeatures in the environment by processing captured images of theenvironment and detecting visual features in the images. For instance,the backup navigation system can cause the image capture device 304 toperiodically capture images of the UAV 300's environment as the UAV 300flies along the flight path. Then, the backup navigation system canprocess the captured images to detect visual features in the images.Within examples, the portion of the environment that is captured in theimage depends on a location of the UAV 300, specifications of the imagecapture device 304, and the orientation of the image capture device 304.

Once the image is captured, the backup navigation system can process theimage to detect unique visual features in the portion of the environmentcaptured in the image. As explained above, identifying a unique visualfeature does not require identification of the type of visual feature(e.g., that a particular feature is a building). Rather, identifying aunique visual feature can involve detecting identifiable characteristicsof the feature, where the identifiable characteristics can be used toidentify the feature at a later time. Such characteristics includedesign features of a structure (e.g., contours, edges, etc.), shape anddimensions of a feature, aesthetic characteristics (e.g., color), amongother example characteristics. Within examples, the backup navigationsystem can detect features using feature detection algorithms such asedge detection algorithms (e.g., Canny, Harris & Stephens, etc.), cornerdetection algorithms (e.g., Shi & Tomasi, FAST, etc.), blob detectionalgorithms (e.g., Laplacian of Gaussian, Difference of Gaussians, etc.)among other feature detection algorithms. Upon detecting these featuresthe backup navigation system can extract unique identifiablecharacteristics of those features using feature extraction algorithmssuch as scale-invariant feature transform (SIFT), speeded up robustfeatures (SURF), BRIEF (Binary Robust Independent Elementary Features),Oriented FAST Rotated BRIEF (ORB), (Fast Retina Keypoint) FREAK, LearnedInvariant Feature Transform (LIFT).

Once a unique visual feature is identified, the backup navigation systemcan then determine a location of the visual feature. Determining thelocation of a visual feature is also referred to herein as “featurelocalization.” The backup navigation system can associate each imagecaptured with a location of the UAV 300 when the image was captured.Although the image capture device 304 can detect visual features from asingle image, e.g., image 420, accurately localizing a feature fromtwo-dimensional images typically involves using two or more images thatshow two or more vantage points of the same features. Accordingly, inorder to determine a respective location for one or more of the detectedfeatures, the image capture device 304 can capture another image of atleast some of the detected features.

As explained above, the image capture device 304 can periodicallycapture images of the UAV 300's environment during the UAV 300's flight.Accordingly, another image of a detected feature can be capturedsubsequent to capturing the image 420. In an embodiment, the imagecapture device 304 can find another vantage point of a feature byextracting visual features from subsequently captured images andcomparing the extracted visual features to the detected visual feature.In an embodiment, comparing an extracted visual feature to the detectedvisual feature can involve grayscale matching, gradient matching,histogram matching, and other feature matching algorithms, such asbrute-force matching and Fast Library for Approximate Nearest Neighbors(FLANN) based matching. Additionally or alternatively, in someembodiments, the comparison may include various image registration ormorphing algorithms.

If a match between an extracted visual feature and a detected visualfeature is detected (i.e., two vantage points of the same feature isdetected in two or more different images), the backup navigation systemcan determine a location of the feature. In particular, the backupnavigation system can use the location and orientation associated witheach image in which the visual feature was detected in order tocalculate the location of the visual feature. For example, the backupnavigation system can use triangulation to determine the location of thevisual feature from the locations associated with the two or more imagesin which the feature was detected. Other methods of calculating afeature's location from the images in which the feature is captured arealso possible. For instance, a feature's location can be calculatedusing methods that employ optimization algorithms.

The backup navigation system can then store the localized feature in amemory of the UAV 300. In particular, the backup navigation system canstore data indicative of identifiable characteristics of the localizedfeature, such as a respective portion of each image that includes thelocalized feature or any other data that can be used to identify thefeature. Additionally, the backup navigation system can store dataindicative of a location of the localized feature. Additionally and/oralternatively, the backup navigation system can store data indicative ofthe conditions under which the images that included the feature werecaptured. Such conditions include operating conditions of the UAV 300and/or the image capture device 304 when the image was captured, such asthe altitude of the UAV 300, the location of the UAV 300, theorientation of the image capture device 304, the resolution of the imagecapture device 304, among other operating conditions.

In an embodiment, the backup navigation system can use the stored datato generate a visual log or map of the flight path. That is, the backupnavigation system can generate a map that includes the features that arelocalized during the UAV 300's flight. In the generated map, eachfeature can be placed at its respective location, and therefore, the mapcan be representative of the environment along the flight path to thedestination. Additionally, the map can be updated each time that thebackup navigation system localizes a feature along the flight path.

In an embodiment, the backup navigation system can operate in thefeature localization mode until the primary navigation system encountersa failure event. That is, the backup navigation system can generateand/or update one or more maps of the environment around the flight pathuntil the primary navigation system encounters the failure event. Withinexamples, the failure event can be any event that affects thefunctionality of the primary navigation system, such as a loss ofconnectivity with remote servers and/or satellites that provide theprimary navigation system with location information, malfunctioning ofdevices of the primary navigation system (e.g., a receiver), among otherfailure events. For instance, the primary navigation system can losesignal from the satellite on which it relies when the UAV 300 enters anarea where satellite signals are attenuated or blocked (e.g., in acity).

As explained above, encountering such failure events can render theprimary navigation system inoperable, and the primary navigation systemmay not be able to navigate the UAV 300 to the destination. Accordingly,to avoid undesirable consequences, the UAV 300, responsive to the UAV300 and/or the primary navigation system encountering a failure event,can determine to return to a safe landing zone. In an example, the safelanding zone can be the location from which the UAV 300 launched, suchas the UAV 300's home base. In another example, the safe landing zonecan be located in an area near the UAV 300's flight path to thedestination.

In an embodiment, the backup navigation system can navigate the UAV 300to the location from which the UAV 300 began its current flight (i.e.,the starting or launch point). In particular, responsive to detectingthe failure event, the backup navigation system can transition fromoperating in the feature localization mode to operating in a UAVlocalization mode in which the system can determine a location of theUAV and can navigate the UAV to the safe zone. In order to navigate theUAV 300 to the safe zone, the backup navigation system can determine acurrent location of the UAV 300. And based on the current location ofthe UAV 300, the system can determine or update a return flight pathfrom the location at which the UAV 300 was located when the failureevent occurred to the safe zone.

In an embodiment, the backup navigation system can use the generated mapof the flight path to the destination (i.e., initial flight path) todetermine the UAV 300's location. To use the map, the backup navigationsystem can capture an image of the UAV 300's current environment, andcan process the captured image to detect visual features of theenvironment. The backup navigation system can then compare the detectedvisual features to features in the map (i.e., features that werelocalized when the backup navigation system was operating in the featurelocalization mode). If the backup navigation system detects arelationship between the detected visual features and the localizedfeatures, the system can determine the location of the detected feature.

In an example, a relationship between the detected feature and thelocalized feature can be that the detected feature and the localizedfeature are the same feature in the environment. That is, the backupnavigation system captured an image of a feature that it has localizedbefore. In such an example, the detected feature has the same locationas the localized features. In another example, the relationship betweenthe features could be a known spatial relationship between the features.For instance, the detected feature could be located at a known distanceaway from the localized feature (e.g., a landmark), and by determiningthe location of the localized feature, the system can determine alocation of the detected feature.

Once the backup navigation system determines a location of a detectedfeature, the system can then use the location of the feature todetermine a location of the UAV 300. As explained above, in practice,determining a location from two-dimensional images can require two knownlocations in the images in order to use the known locations to calculatethe unknown location. Therefore, in an embodiment, the backup navigationsystem can determine a respective location of two or more detectedfeatures in the captured image in order to determine a location of theUAV 300. By way of example, the backup navigation system can determinethat two identified features match two respective localized features inthe map, and can determine the location of the two identified featuresbased on the locations of the respective localized features in the map.Then, the system can calculate the location of the UAV 300 using thelocations of the two identified features. For instance, the system canuse the two known locations to triangulate the location of the UAV 300.

In another embodiment, the backup navigation system can determine thelocation of the UAV from the location of a single detected feature. Inparticular, the system can use the location of the detected feature andinformation indicative of the orientation of the UAV to calculate anapproximate location of the UAV. In an example, to perform thiscalculation, the system can assume that the height above ground level(AGL) of the image capture device (at the current location) is equal tothe average AGL of the image capture device when capturing all of theimages that were used to localize the feature. Given the average AGL,the current orientation of the UAV, and the field-of-view of the imagecapture device, a 3D volume above the detected feature within which theUAV should be located can be determined.

Once the backup navigation system has determined the current location ofthe UAV 300, the system can determine a flight path to the safe zone.This flight path is also referred to herein as a return flight path. Inan embodiment, the backup navigation system can determine a returnflight path that is identical or substantially similar to the initialflight path. A return flight path that is substantially similar to theinitial flight path is a flight path where at least a given minimumnumber of the localized features are within the FOV of the image capture304 throughout the flight.

In an example, the backup navigation system can use the generated map todetermine the minimum number of localized features that the system wouldneed to detect in order to navigate to the safe zone. The system canthen determine the return flight path such that the minimum number oflocalized features is within the FOV of the image capture device 304when the UAV 300 is flying along the return flight path.

In another example, the system can use the generated map to identify thelocalized features that the system would need to detect in order toreach the safe zone. The system can then determine the return flightpath such that the localized features are within the FOV of the imagecapture device 304 when the UAV 300 is flying along the return flightpath. By determining such a return flight path that is identical orsubstantially similar to the initial flight path, the backup navigationsystem can use the generated map of the initial flight path to navigatealong the return flight path.

FIGS. 4A-4F depict a scenario 400 that illustrates operation of thebackup navigation system, according to exemplary embodiments. In thescenario 400, the UAV 300 is flying from a starting point (e.g., a homebase) to a destination. Before the UAV 300 begins the flight, theprimary navigation system of the UAV 300 can determine a flight pathfrom the starting point to the destination. Once the primary navigationsystem determines the flight path, the UAV 300 can fly along thedetermined flight path to the destination.

FIG. 4A illustrates the UAV 300's flight to a destination, according toan exemplary embodiment. In particular, the UAV 300 is flying along aflight path 480 that is determined by the primary navigation system(before the UAV 300 began the flight). During the flight, the primarynavigation system is tasked with navigating the UAV 300 along thedetermined flight path. For instance, as illustrated in FIG. 4A, theprimary navigation system has navigated the UAV 300 to reach a position“A” along the flight path 480. Further, the UAV 300 has not encountereda failure event, and therefore, the primary navigation system is stillfunctioning. As such, the backup navigation system does not need toprovide the UAV 300 with navigational updates.

However, as explained above, the backup navigation system can operate inthe feature localization mode while the primary navigation system isnavigating the UAV 300. In particular, the backup navigation system cangenerate and/or update a map of the UAV 300's environment. For instance,as illustrated in FIG. 4A, in the scenario 400, the flight path 480passes over the city 402, where the city 402 includes structures 404,406, and 412, among other structures and features. Accordingly, the UAV300's environment in scenario 400 can be the city 402.

In an embodiment, the backup navigation system can detect features ofthe city 402 by capturing images of the city 402. For example, when theUAV 300 is at position A, the image capture device 304 can capture animage of the city 402. The portion of the city 402 that is captured inthe image depends on a location of the UAV 300, specifications of theimage capture device 304, and the orientation of the image capturedevice 304. As illustrated in FIG. 4A, since the UAV 300 is at positionA, the image capture device 304 has a FOV of 90 degrees (i.e., areabetween lines 408 a and 408 b), and the image capture device 304 isoriented perpendicular to the UAV 300, the portion of the environmentcaptured by an image is area 410 of the city 402. As shown in FIG. 4A,the buildings 404 and 412 are located in the area 410 of the city 402.

FIG. 4B illustrates a representation of an image 420 of the area 410,according to an exemplary embodiment. In this scenario, theconfiguration of the image capture device 304 is such that image capturedevice 304 captures a top view image of the environment (e.g., the city402). Accordingly, the vantage point of the image 420 is a top view(i.e., bird's eye view) of the area 410. The features of the area 410that are captured by the image capture device 304 can depend on thespecifications of the image capture device 304 (e.g., resolution, lensspecifications, etc.) and on the altitude of the UAV 300 when the image420 was captured. For instance, the image capture device 304 can capturestructures (e.g., buildings), terrain, water bodies, and other featuresof the area 410. As illustrated in FIG. 4B, the image 420 includes anopen area 422 (e.g., a park or parking lot), structures 422, 426, and428, among other captured features. For the sake of simplicity, theimage 420 includes large features such as structures or open areas, butin practice, other less prominent features (e.g., trees or signs) canalso be captured in the image.

Once the image 420 is captured, the backup navigation system can processthe image 420 to detect unique visual features in the area 410, such asstructures or open areas. As explained above, identifying a uniquevisual feature does not require identification of the type of visualfeature (e.g., that a particular feature is a building). Rather,identifying a unique visual feature can involve detecting identifiablecharacteristics of the feature, where the identifiable characteristicscan be used to identify the feature at a later time. Suchcharacteristics include design features of a structure (e.g., contours,edges, etc.), shape and dimensions of a feature, aestheticcharacteristics (e.g., color), among other example characteristics.Within examples, the backup navigation system can detect features usingfeature detection algorithms such as edge detection algorithms (e.g.,Canny, Harris & Stephens, etc.), corner detection algorithms (e.g., Shi& Tomasi, Laplacian of Gaussian, etc.), among other feature detectionalgorithms.

For example, the backup navigation system can process the image 420 todetect features in the area 410. By way of example, by processing theimage 420 the backup navigation system can detect structures 424, 426,and 428, and open area 422, among other features. The backup navigationsystem can then extract data that represents the unique identifiablecharacteristics of the features (e.g., patch of the image in thevicinity of the feature, histogram of a patch of the image centered onthe feature, etc.). The backup navigation system can then store thedetected features in a memory of the UAV 300.

As explained above, more than one vantage point of a feature can be usedto calculate a location of the feature. Accordingly, the image capturedevice 304 can capture another image of the city 402 in order tolocalize one or more visual features in the city 402. As explainedabove, the image capture device 304 can be configured to periodicallycapture images of the UAV 300's environment. Accordingly, the backupnavigation system can compare features that are captured in a firstimage to features that are captured in subsequent images to find othervantage points of the features captured in the first image. If thebackup navigation system detects another vantage point of a feature in asubsequent image, the system can use the vantage point from the firstimage and the vantage point from the subsequent image to determine thelocation of the feature.

FIG. 4C depicts the UAV 300 at position “B” along the UAV 300's flightpath 480. At position B, the image capture device 304 can capture animage of the city 402. As illustrated in FIG. 4C, the image capturedevice 304 can capture an image of an area 430 that is within the FOV ofthe image capture device 304 when the UAV 300 is at position B. Thebackup navigation system can process the image 432 to extract featuresthat are located in the area 430. The system can then compare theextracted features to features that were detected in the image 420 inorder to determine whether the image 432 includes another vantage pointof the features detected in image 420.

FIG. 4D depicts an image 432 of the area 430, according to an exemplaryembodiment. Like image 420 of FIG. 4B, the image 432 captures structuresand open areas that are located in the area 430. As illustrated in FIG.4D, the image 432 includes buildings, such as buildings 438 and 436,parking lots, such as parking lot 440, and other features, such as awater fountain 434. Further, the backup navigation system can processthe image 432 to identify unique visual features of the area 430. Forinstance, processing the image can identify the features 436, 438, and442. As illustrated in FIG. 4D, the detected features are marked by adotted line. The backup navigation system can then compare one or morevisual features detected in the image 420 to one or more visual featuresdetected in the image 432. By way of example, by performing thecomparison, the backup navigation system can detect that the feature 426in the image 420 and the feature 438 in the image 432 correspond to oneanother. That is, the feature 426 and the feature 438 are bothindicative of the same building, building 412. As illustrated in FIGS.4A and 4C, building 412 is in the FOV of the image capture device 304when the UAV 300 is at position A and when the UAV 300 is at position B.Accordingly, the building 412 can be captured in the images taken atboth position A and position B.

The backup navigation system can then use the two vantage points ofbuilding 412 to determine a location of the building 412. In anembodiment, the backup navigation system can use the respective locationinformation that is associated with the images 420 and 432 to determinethe location of the building 412. In an example, the location of thebuilding 412 can be determined using triangulation. Furthermore, morethan two images can be used to determine a location of a featuredetected in the images.

The backup navigation system can then store the localized feature in amemory of the UAV 300. Additionally, the backup navigation system canlocalize other features located in the two images 420, 432. The backupnavigation system can also capture other images of the city 402 andlocalize other features of the city 402. The backup navigation systemcan then use the localized features to generate and/or update a map ofthe flight path 480. For example, the map can include the building 412that has been localized. The map can indicate the location of thebuilding 412 in the city 402 and can also indicate uniquecharacteristics of the building 412 that can be used to identify thebuilding 412.

As explained above, the backup navigation system can operate in thefeature localization mode until the UAV 300 detects a failure event thatcan affect the operability of the primary navigation system. Responsiveto detecting the failure event, the backup navigation system cantransition from operating in the feature localization mode to operatingin the UAV localization mode.

FIG. 4E illustrates the UAV 300 that has encountered a failure event,according to an example embodiment. In particular, at the position Calong the flight path, the primary navigation system encounters afailure event. And responsive to detecting the failure event, the backupnavigation system can transition from operating in the featurelocalization mode to operating in the UAV localization mode. Asexplained above, in the UAV localization mode, the backup navigationsystem can navigate the UAV 300 to a safe landing zone. In the scenario400, the backup navigation system can determine to navigate the UAV 300from position C and back to the starting point.

In an embodiment, the backup navigation system can determine the UAV300's current position using the map of the flight path 480. Inparticular, the backup navigation system can cause the image capturedevice 304 to capture an image of the UAV 300's environment at positionC. As illustrated in FIG. 4E, an area 450 of the city 402 is within theFOV of the image capture device 304 when the UAV 300 is at position C.

FIG. 4F illustrates an image 452 of the area 450, according to anexample embodiment. Within examples, the backup navigation system canprocess the image 452 in order to identify visual features that arelocated in the area 450. As illustrated in FIG. 4F, processing the image452 can detect the features 454, 456, and 458, among other features. Thebackup navigation system can then compare the detected features to thelocalized features in order to detect a relationship between thedetected features and the localized features. For example, the backupnavigation system can compare the features 454, 456, and 458 to thelocalized features 424, 426, and 428 in FIG. 4B and to the localizedfeatures 434, 436, 438, 440, and 442 in FIG. 4D.

In an example, the backup navigation system can determine the features454 and 456 in FIG. 4F are respectively the features 438 and 436 of FIG.4D. Accordingly, the backup navigation system can determine the locationof the features 454 and 456 since the backup navigation system hadpreviously calculated their locations when operating in the featurelocalization mode. The backup navigation system can then calculate basedon the locations of the features 454 and 456 the current location of theUAV 300. For instance, the backup navigation system can usetriangulation to calculate the position of the UAV 300.

Once the backup navigation system determines the current position of theUAV 300, the backup navigation system can determine at least a portionof the flight path back to the safe zone. For instance, as illustratedin FIG. 4E, the backup navigation system can determine a portion of thereturn flight path 482. Then once the backup navigation system detectsanother relationship between a detected visual feature and a localizedfeature, the system can determine another portion of the return flightpath.

The example scenario 400 depicted in FIGS. 4A-4F and the accompanyingdescription herein is for illustrative purposes only and should not beconsidered limiting. In an implementation, the image capture device 304may have a different orientation. For instance, if the UAV 300 is flyingat a lower altitude (e.g., near the buildings of the city 402), theorientation of the image capture device 304 can be forward-facing inorder to capture images of the structures in the city 402. The backupnavigation system can then detect features in the city 402 by detectingcharacteristics of the features. By way of example, the backupnavigation system can detect a spire 414 of the building 404 as acharacteristic of the building 404. Other example orientations of theimage capture device 304 are possible. As such, other example vantagepoints of the UAV 300's environment are also possible.

In another implementation, the UAV 300 can preload a map beforebeginning a flight to a destination. For instance, the map can be of theenvironment of a flight path, where the map includes locations and/orimage data of landmarks that are located in the environment of theflight path. The backup navigation system operating in the featurelocalization mode during the flight can update the map with additionalfeatures of the environment that are detected during the flight.Additionally, the map can be used by the UAV to determine a location ofthe UAV when the backup navigation system is operating in the UAVlocalization mode. In an example, the UAV 300 can retrieve the map froma local and/or remote server. In another example, a dispatch system canprovide the map to the UAV 300 with the instructions to fly to adestination.

In an embodiment, the preloaded map can be a spatial and temporal map.That is, in addition to being associated with an area, the map can alsobe associated with a time and/or date. Such a map can also be referredto as a “spatio-temporal” map. For instance, the time and/or date can bea time and/or date of the UAV 300's flight to a destination. By way ofexample, the UAV 300 or the dispatch system can determine temporalcharacteristics of the UAV's 300 of a destination, and could select amap whose temporal association encompasses the temporal characteristicsof the UAV 300's flight. In an example, the map can be associated with atemporal range (e.g., a time and/or date range), and if the flight fallsin that time range, could select that map. This is useful because theidentifying characteristics of the environment could change depending onthe time of the day or date. For example, certain features may not bevisible at night, and other features may not be visible during thewinter (e.g., due to snowfall or other environmental conditions). Otherexamples of dynamic features include shadows (which depend on time anddate) and colors of natural features (which can depend on the season).

In another implementation, the backup navigation system can detectfeatures in environments other than a city. Such environments can beindoor (e.g., a warehouse) or outdoor environments. In some examples,the UAV 300 can encounter more than one environment during its flight.For instance, a portion of the flight can be over a city, and anotherportion can be over a countryside area.

FIG. 5 illustrates an image 500 of an area 502 that has a hilly ormountainous terrain, according to an exemplary embodiment. In anexample, the image 500 is captured by an image capture device of a UAVthat is flying over the area 502 en route to a destination. Withinexamples, the backup navigation system of the UAV can process the image500 to detect features of the area 502. For example, the backupnavigation system can detect features 504 and 506 of the area 502. Asillustrated in FIG. 5, the features 504, 506 can be features of theterrain that have unique characteristics. For instance, thecharacteristics of a hilly terrain feature can be a slope, a ridge, avalley, a hill, a depression, a saddle, a cliff, among othercharacteristics. As also illustrated in FIG. 5, the backup navigationsystem can mark the detected features 504, 506 with a dotted line thatencompasses the feature. The backup navigation system can then determinethe unique characteristics of the features 504, 506 and can use theunique characteristics to generate and/or update a map of the area 502.Similarly, the backup navigation system can detect a water body 508, andcan store a location of the water body 508 in a memory of the UAV. Otherexamples of detected features are possible.

V. Map Database

Also disclosed herein is a system that can generate and/or update adatabase of maps and/or features that can be used by a UAV fornavigation. In an embodiment, the system can include a plurality of UAVsthat can be deployed to perform tasks that involve flying to adestination. The system can also include a server that can beresponsible for generating and/or updating the maps based on data fromthe plurality of UAVs. In some examples, the server can be responsiblefor coordinating the flights of the plurality of UAVs.

In an embodiment, the plurality of UAVs can share their local “featuredatabases” (i.e., databases of local flights logs and maps) with oneanother and/or with the server. A UAV or server, upon receiving adatabase from another UAV, can merge the received database into theirrespective local database using the same feature matching techniquesused by the backup navigation system when operating in the UAVlocalization mode. In an example, the server can use the databasesreceived from the plurality of UAVs to generate a global map of theareas in which the UAVs are typically deployed. Later, a UAV, beforebeing deployed to perform a task, can preload the global map to theUAV's memory. As explained above, the backup navigation system can usethe preloaded global map to replace the navigational functionality ofthe UAV upon failure of the primary navigation system and/or to checkwhether the primary navigation system is functioning properly.

In an embodiment, the server can generate and/or update spatio-temporalmaps of the environment. A spatio-temporal map is associated withtemporal and/or environmental data. In particular, the temporal and/orenvironmental data can be indicative of temporal data and/orenvironmental data at the time that the images used to generate the mapwere captured. The temporal data can include a time and/or date at whichthe images were captured. For instance, a map can be associated with atime and/or date range. The time range can be on the order of second,minutes, or hours. And the date range can be on the order of days,weeks, months, or years. As for the environmental data, such data can beindicative of meteorological conditions such as the weather of anenvironment when the image of the environment was captured.

In an embodiment, based on the flight path of a UAV, the server canpreload a map that is associated with the flight path. In particular,the map can be associated with an area that the flight path willtraverse and at least one of (i) temporal data of the flight path (e.g.,time of day of the flight, date of the flight) and (ii) environmentaldata of the flight (e.g., weather conditions of the area that the flightpath will traverse).

In some examples, the UAVs can share with other UAVs or the server theimages that they captured during their flight. The captured images caninclude metadata indicative of geolocation data and data indicative ofthe status of the UAV and/or the image capture device when the image wascaptured. For instance, the metadata can include data indicative of theorientation of the image capture device when the image was captured.

In another embodiment, when the backup navigation system of a UAV of theplurality transitions from operating in the feature localization mode tooperating in the UAV localization mode, the server can send the UAV amap of the area in which the UAV is located. In an example, the map sentto the UAV can further depend on temporal data and/or environmental dataassociated with the UAVs flight.

Such a system can allow a UAV, which for various reasons (e.g., fuelconsiderations), needs to lands in a safe zone in area for which the UAVhas not generated a local flight log or map (i.e., the UAV has nottraversed the area) to navigate to the safe zone. The system can alsoallow the UAV to traverse to a new area that the UAV has not traversedto previously.

IV. Illustrative UAV Components

FIG. 6 is a simplified block diagram illustrating components of a UAV600, according to an example implementation. UAV 600 may take the formof, or be similar in form to, one of the UAVs 100, 120, 140, 160, and180 described in reference to FIGS. 1A-1E.

UAV 600 may include various types of sensors, and may include acomputing device configured to provide the functionality describedherein. In the illustrated implementation, the sensors of UAV 600include an inertial measurement unit (IMU) 602, ultrasonic sensor(s)604, and a navigation system 606, among other possible sensors andsensing systems. In the illustrated implementation, UAV 600 alsoincludes one or more processors 608. A processor 608 may be ageneral-purpose processor or a special purpose processor (e.g., digitalsignal processors, application specific integrated circuits, etc.). Theone or more processors 608 can be configured to executecomputer-readable program instructions 612 that are stored in the datastorage 610 and are executable to provide the functionality of a UAVdescribed herein.

The data storage 610 may include or take the form of one or morecomputer-readable storage media that can be read or accessed by at leastone processor 608. The one or more computer-readable storage media caninclude volatile and/or non-volatile storage components, such asoptical, magnetic, organic or other memory or disc storage, which can beintegrated in whole or in part with at least one of the one or moreprocessors 608. In some implementations, the data storage 610 can beimplemented using a single physical device (e.g., one optical, magnetic,organic or other memory or disc storage unit), while in otherimplementations, the data storage 610 can be implemented using two ormore physical devices.

As noted, the data storage 610 can include computer-readable programinstructions 612 and perhaps additional data, such as diagnostic data ofthe UAV 600. As such, the data storage 610 may include programinstructions 612 to perform or facilitate some or all of the UAVfunctionality described herein. For instance, in the illustratedimplementation, program instructions 612 include a navigation module614.

A. Sensors

In an illustrative implementation, IMU 602 may include both anaccelerometer and a gyroscope, which may be used together to determinean orientation of the UAV 600. In particular, the accelerometer canmeasure the orientation of the vehicle with respect to earth, while thegyroscope measures the rate of rotation around an axis. IMUs arecommercially available in low-cost, low-power packages. For instance, anIMU 602 may take the form of or include a miniaturizedMicroElectroMechanical System (MEMS) or a NanoElectroMechanical System(NEMS). Other types of IMUs may also be utilized.

An IMU 602 may include other sensors, in addition to accelerometers andgyroscopes, which may help to better determine position and/or help toincrease autonomy of the UAV 600. Two examples of such sensors aremagnetometers and pressure sensors. In some implementations, a UAV mayinclude a low-power, digital 3-axis magnetometer, which can be used torealize an orientation independent electronic compass for accurateheading information. However, other types of magnetometers may beutilized as well. Other examples are also possible. Further, note that aUAV could include some or all of the above-described inertia sensors asseparate components from an IMU.

UAV 600 may also include a pressure sensor or barometer, which can beused to determine the altitude of the UAV 600. Alternatively, othersensors, such as sonic altimeters or radar altimeters, can be used toprovide an indication of altitude, which may help to improve theaccuracy of and/or prevent drift of an IMU.

In a further aspect, UAV 600 may include one or more sensors that allowthe UAV to sense objects in the environment. For instance, in theillustrated implementation, UAV 600 includes ultrasonic sensor(s) 604.Ultrasonic sensor(s) 604 can determine the distance to an object bygenerating sound waves and determining the time interval betweentransmission of the wave and receiving the corresponding echo off anobject. A typical application of an ultrasonic sensor for UAVs or IMUsis low-level altitude control and obstacle avoidance. An ultrasonicsensor can also be used for vehicles that need to hover at a certainheight or need to be capable of detecting obstacles. Other systems canbe used to determine, sense the presence of, and/or determine thedistance to nearby objects, such as a light detection and ranging(LIDAR) system, laser detection and ranging (LADAR) system, and/or aninfrared or forward-looking infrared (FLIR) system, among otherpossibilities.

In some implementations, UAV 600 may also include one or more imagingsystem(s). For example, one or more still and/or video cameras may beutilized by UAV 600 to capture image data from the UAV's environment. Asa specific example, charge-coupled device (CCD) cameras or complementarymetal-oxide-semiconductor (CMOS) cameras can be used with UAVs. Suchimaging sensor(s) have numerous possible applications, such as obstacleavoidance, localization techniques, ground tracking for more accuratenavigation (e.g., by applying optical flow techniques to images), videofeedback, and/or image recognition and processing, among otherpossibilities.

UAV 600 may also include a navigation system 606. The navigation system606 can be configured to provide navigational functionality to the UAV600. For instance, the navigation system 606 can be a GPS receiver thatcan be configured to provide data that is typical of well-known GPSsystems, such as the GPS coordinates of the UAV 600. Such GPS data maybe utilized by the UAV 600 for various functions. As such, the UAV mayuse its GPS receiver to help navigate to a destination, as indicated, atleast in part, by the GPS coordinates of the destination. Other examplesare also possible.

A. Navigation and Location Determination

The navigation system 606 can include device (e.g., receivers, sensors,etc.) of the primary navigation system and the backup navigation system.For example, where the primary navigation system is a system that relieson GPS, the navigation system 606 can include a GPS receiver that isconfigured to receive coordinates from a GPS satellite. Additionallyand/or alternatively, the navigation system 606 can include an imagecapture device that can be configured to capture images of the UAV 600'senvironment near the UAV 600's flight path.

Further, a navigation module 614 can utilize the devices of thenavigation system 606 to provide the UAV 600 with navigationalfunctionality (e.g., allows the UAV 600 to move about its environmentand reach a desired location). To navigate the UAV 600, the navigationmodule 614 can control the altitude and/or direction of flight bycontrolling the mechanical features of the UAV 600 that affect flight(e.g., its rudder(s), elevator(s), aileron(s), and/or the speed of itspropeller(s)). And in order to navigate the UAV 600 to a targetlocation, the navigation module 614 can implement various navigationtechniques, such as map-based navigation, for instance. For instance, inconjunction with coordinate data received from the primary navigationsystem of the navigation system 606, the UAV 600 can use a map of itsenvironment, which can then be used to navigate to a particular locationon the map.

In some implementations, the navigation module 614 can navigate using atechnique that relies on waypoints. In particular, waypoints are sets ofcoordinates that identify points in physical space. For instance, anair-navigation waypoint may be defined by a certain latitude, longitude,and altitude. Accordingly, the navigation module 614 can cause UAV 600to move from waypoint to waypoint, in order to ultimately travel to afinal destination (e.g., a final waypoint in a sequence of waypoints).

In a further aspect, the navigation module 614 and/or other componentsand systems of the UAV 600 may be configured for “localization” to moreprecisely navigate to the scene of a target location. More specifically,it can be desirable in certain situations for the UAV 600 to be within athreshold distance of the target location where a payload 628 is beingdelivered by the UAV 600 (e.g., within a few feet of the targetdestination). To this end, the UAV 600 can use a two-tiered approach inwhich it uses a more-general location-determination technique tonavigate to a general area that is associated with the target location,and then use a more-refined location-determination technique to identifyand/or navigate to the target location within the general area.

For example, the UAV 600 can navigate to the general area of a targetdestination where a payload 628 is being delivered using waypointsand/or map-based navigation. The UAV 600 can then switch to a mode inwhich it utilizes a localization process to locate and travel to a morespecific location. For instance, if the UAV 600 is to deliver a payloadto a user's home, the UAV 600 may need to be substantially close to thetarget location in order to avoid delivery of the payload to undesiredareas (e.g., onto a roof, into a pool, onto a neighbor's property,etc.). However, a GPS signal can only get the UAV 600 so far (e.g.,within a block of the user's home). A more preciselocation-determination technique can then be used to find the specifictarget location.

Various types of location-determination techniques can be used toaccomplish localization of the target delivery location once the UAV 600has navigated to the general area of the target delivery location. Forinstance, the UAV 600 can be equipped with one or more sensory systems,such as, for example, ultrasonic sensors 604, infrared sensors (notshown), and/or other sensors, which can provide input that thenavigation module 614 utilizes to navigate autonomously orsemi-autonomously to the specific target location.

As another example, once the UAV 600 reaches the general area of thetarget delivery location (or of a moving subject such as a person ortheir mobile device), the UAV 600 can switch to a “fly-by-wire” modewhere it is controlled, at least in part, by a remote operator, who cannavigate the UAV 600 to the specific target location. To this end,sensory data from the UAV 600 can be sent to the remote operator toassist them in navigating the UAV 600 to the specific location.

As yet another example, the UAV 600 may include a module that is able tosignal to a passer-by for assistance in reaching the specific targetdelivery location. For example, the UAV 600 may display a visual messagerequesting such assistance in a graphic display, with the visual messagepossibly indicating the specific target delivery location, among otherpossibilities. In another example, the UAV 600 can play an audio messageor tone through speakers to indicate the need for such assistance, withthe audio message or tone possibly indicating the specific targetdelivery location, among other possibilities. In practice, such afeature can be useful in a scenario in which the UAV is unable to usesensory functions or another location-determination technique to reachthe specific target location. However, this feature is not limited tosuch scenarios.

In some implementations, once the UAV 600 arrives at the general area ofa target delivery location, the UAV 600 may utilize a beacon from auser's remote device (e.g., the user's mobile phone) to locate theremote device, person, or location. Such a beacon may take variousforms. As an example, consider the scenario where a remote device, suchas the mobile phone of a person who requested a UAV delivery, is able tosend out directional signals (e.g., via an RF signal, a light signaland/or an audio signal). In this scenario, the UAV 600 may be configuredto navigate by “sourcing” such directional signals—in other words, bydetermining where the signal is strongest and navigating accordingly. Asanother example, a mobile device can emit a frequency, either in thehuman range or outside the human range, and the UAV 600 can listen forthat frequency and navigate accordingly. As a related example, if theUAV 600 is listening for spoken commands, then the UAV 600 could utilizespoken statements, such as “I'm over here!” to source the specificlocation of the person requesting delivery of a payload.

In an alternative arrangement, a navigation module may be implemented ata remote computing device, which communicates wirelessly with the UAV600. The remote computing device may receive data indicating theoperational state of the UAV 600, sensor data from the UAV 600 thatallows it to assess the environmental conditions being experienced bythe UAV 600, and/or location information for the UAV 600. Provided withsuch information, the remote computing device may determine altitudinaland/or directional adjustments that should be made by the UAV 600 and/ormay determine how the UAV 600 should adjust its mechanical features(e.g., its rudder(s), elevator(s), aileron(s), and/or the speed of itspropeller(s)) in order to effectuate such movements. The remotecomputing device may then communicate such adjustments to the UAV 600 soit can move in the determined manner.

B. Communication Systems

In a further aspect, the UAV 600 can include one or more communicationsystems 618. In some examples, the UAV 600 does not include thecommunication systems 618. In other examples, the UAV 600 can functionwithout the communication system 618 if for some reason thecommunication systems 618 were rendered inoperable. Within examples, thecommunications systems 618 may include one or more wireless interfacesand/or one or more wireline interfaces, which allow the UAV 600 tocommunicate via one or more networks. Such wireless interfaces mayprovide for communication under one or more wireless communicationprotocols, such as Bluetooth, WiFi (e.g., an IEEE 902.11 protocol),Long-Term Evolution (LTE), WiMAX (e.g., an IEEE 902.16 standard), aradio-frequency ID (RFID) protocol, near-field communication (NFC),and/or other wireless communication protocols. Such wireline interfacesmay include an Ethernet interface, a Universal Serial Bus (USB)interface, or similar interface to communicate via a wire, a twistedpair of wires, a coaxial cable, an optical link, a fiber-optic link, orother physical connection to a wireline network.

In some implementations, a UAV 600 may include communication systems 618that allow for both short-range communication and long-rangecommunication. For example, the UAV 600 may be configured forshort-range communications using Bluetooth and for long-rangecommunications under a CDMA protocol. In such an implementation, the UAV600 may be configured to function as a “hot spot;” or in other words, asa gateway or proxy between a remote support device and one or more datanetworks, such as a cellular network and/or the Internet. Configured assuch, the UAV 600 may facilitate data communications that the remotesupport device would otherwise be unable to perform by itself.

For example, the UAV 600 may provide a WiFi connection to a remotedevice, and serve as a proxy or gateway to a cellular service provider'sdata network, which the UAV might connect to under an LTE or a 3Gprotocol, for instance. The UAV 600 could also serve as a proxy orgateway to a high-altitude balloon network, a satellite network, or acombination of these networks, among others, which a remote device mightnot be able to otherwise access.

C. Power Systems

In a further aspect, the UAV 600 may include power system(s) 620. Thepower system 620 may include one or more batteries for providing powerto the UAV 600. In one example, the one or more batteries may berechargeable and each battery may be recharged via a wired connectionbetween the battery and a power supply and/or via a wireless chargingsystem, such as an inductive charging system that applies an externaltime-varying magnetic field to an internal battery.

VI. Example Server

FIG. 7 illustrates a schematic diagram of a server 700, according to anexample implementation. The server 700 includes one or more processor(s)702 and data storage 704, such as a non-transitory computer readablemedium. Additionally, the data storage 704 is shown as storing programinstruction 706, which may be executable by the processor(s) 702.Further, the server 700 also includes a communication interface 708.Note that the various components of server 700 may be arranged andconnected in any manner.

Moreover, the above description of processor(s) 702, data storage 704,and communication interface 708, may apply to any discussion relating tothe respective component being used in another system or arrangements.For instance, as noted, FIG. 7 illustrates processors, data storage, anda communication interface as being incorporated in another arrangement.These components at issue may thus take on the same or similarcharacteristics (and/or form) as the respective components discussedabove in association with FIG. 7. However, the components at issue couldalso take on other characteristics (and/or form) without departing fromthe scope of the disclosure.

In practice, a server may be any program and/or device that providesfunctionality for other programs and/or devices (e.g., any of theabove-described devices), which could be referred to as “clients”.Generally, this arrangement may be referred to as a client-server model.With this arrangement, a server can provides various services, such asdata and/or resource sharing with a client and/or carrying outcomputations for a client, among others. Moreover, a single server canprovide services for one or more clients and a single client can receiveservices from one or more servers. As such, servers could take variousforms (currently known or developed in the future), such as a databaseserver, a file server, a web server, and/or an application server, amongother possibilities.

Generally, a client and a server may interact with one another invarious ways. In particular, a client may send a request or aninstruction or the like to the server. Based on that request orinstruction, the server may perform one or more operations and may thenrespond to the client with a result or with an acknowledgement or thelike. In some cases, a server may send a request or an instruction orthe like to the client. Based on that request or instruction, the clientmay perform one or more operations and may then respond to the serverwith a result or with an acknowledgement or the like. In either case,such communications between a client and a server may occur via a wiredconnection or via a wireless connection, such as via a network forinstance.

VII. Illustrative Methods

FIG. 8 is a flowchart of method 800, in accordance with an exampleembodiment. Method 800 can be carried out by a UAV, such as the UAV 600illustrated in FIG. 6. In particular, the UAV 600 or a controllerthereof can execute software to carry out method 800. Additionallyand/or alternatively, the method 800 can be carried out by a dispatchsystem, such as local dispatch system 212 or central dispatch system 210of FIG. 2.

Method 800 can begin at block 802, where the method 800 involvesoperating an unmanned aerial vehicle (UAV) to begin a flight along afirst flight path to a destination, where the UAV uses a primarynavigation system to navigate to the destination, such as discussedabove in the context of at least FIGS. 4A-4F and FIG. 6. For example,the UAV can be any of the UAVs illustrated in FIGS. 1A-1E, 3, and 4A-4F.As also discussed elsewhere herein, the primary navigation system can bea system that relies on GPS, and therefore, the system can include a GPSreceiver.

As illustrated at block 804, the method 800 further involves operatingan image capture device coupled to the UAV and having a field-of-view(FOV) of an environment of the UAV to capture a first set of images asthe UAV travels along the first flight path. As explained elsewhereherein, in an embodiment, the image capture device can be a camera thathas a 90 degree FOV. Further, the image capture device can be configuredto periodically capture images of the UAV's environment as the UAVtraverses a determined flight path to the destination.

As illustrated at block 806, the method 800 further involves analyzingthe images to localize visual features within the environment. Analyzingthe images can involve using feature detection algorithms to identify ordetect features in the image. Then the detected features could belocalized, which involves determining a location of the detectedfeatures in the environment.

As illustrated at block 808, the method 800 further involves generatinga local flight log for the first flight path, which comprises locationinformation and image data for the detected features. The flight log ormap can include location information associated with one or morefeatures that are detected during the UAV's flight. Additionally and/oralternatively, the flight log can include image data that can be used toidentify the feature. In an example, the image data can be a capturedimage of the feature. Additionally and/or alternatively, the image datacan include identifying characteristics that can be used to identify thefeature. For instance, the identifying characteristics can be a designcharacteristics of the feature, shape and dimensions of the feature,and/or aesthetic characteristics of the feature. Therefore, the map canbe used to identify a particular feature, and once the feature isidentified, a location of the feature can be determined.

As illustrated at block 810, the method 800 involves detecting a failureevent involving the primary navigation system. As explained above, thefailure event can be any event that can affect the operability of theprimary navigation system, such as a receiver malfunction, a loss ofsatellite signal, among other examples.

As illustrated at block 812, the method 800 further involves responsiveto detecting the failure event, capturing at least one post-failureimage using the image capture device. That is, responsive to detectingthe failure event, the image capture device can capture an image of theUAV's environment, e.g., as part of operating in the UAV localizationmode.

As illustrated at block 814, the method 800 further involvesidentifying, in the at least one post-failure image, one or more visualfeatures within the environment. In line with the discussion above, thebackup navigation system can process the post-failure image in order todetect features of the environment that were captured in the image. Inan example, the backup navigation system can compare the featuresextracted from the post-failure image to localized features that arestored locally in a memory of the UAV. The localized features can befeatures that the backup navigation system localized when operating inthe feature localization mode or can be features that have a knownlocation in the map (e.g., a landmark). In some examples, the system canidentify the feature (e.g., a landmark) but may not have locationinformation of the feature.

As illustrated at block 816, the method further involves determining acurrent location of the UAV by determining a relationship between afirst identified visual feature and a first localized visual feature. Inan example, the relationship between the first identified visual featureand the first localized feature can be one of correspondence. Forinstance, the system can determine that the first identified visualfeature was previously localized, and therefore, the system can detectthat the first identified visual feature matches or is identical to thefirst localized feature. Other relationships between an identified and alocalized feature are possible. In an example, the backup navigationsystem can match at least one of the identified or extracted features toat least one of the localized features. Then, the system can use theknown location of the at least one localized feature to calculate thelocation of the UAV, e.g., by using a triangulation algorithm.

FIG. 9 is a flowchart of method 900, in accordance with an exampleembodiment. Method 900 can be carried out by a UAV, such as the UAV 600illustrated in FIG. 6. In particular, the UAV 600 or a controllerthereof can execute software to carry out method 900. Additionallyand/or alternatively, the method 900 can be carried out by a dispatchsystem, such as local dispatch system 212 or central dispatch system 210of FIG. 2.

Method 900 can begin at block 902, where the method 900 involves acontroller of a UAV configured to operate the UAV to begin a flightalong a first flight path to a destination. As explained herein, thefirst flight path can be determined by the primary navigation systembefore the UAV begins the flight. Additionally, the primary navigationsystem can be responsible for providing the controller with navigationalupdates that allow the controller to navigate the UAV.

As illustrated at block 904, the method 900 can further involve thecontroller configured to operate a backup navigation system in a featurelocalization mode in which the controller: (i) causes an image capturedevice to capture a first set of images along the first flight path, and(ii) analyzes the images to localize a set of visual features within anenvironment, where each localized visual feature is associated with arespective geolocation, and where the set of localized features isstored in a memory location. Within examples, a geolocation can be anytype of identification or estimation of the real-world geographiclocation of a feature.

As illustrated at block 906, the method 900 can further involve thecontroller configured to detect a failure event involving a primarynavigation system of the UAV.

As illustrated at block 908, the method 900 can further involve thecontroller configured to, responsive to detecting the failure event,operate the backup navigation system in a UAV localization mode in whichthe controller: (i) causes the image capture device to capture at leastone image of the environment, (ii) identifies in the at least one imagevisual features within the environment, and (iii) compares theidentified visual features to the set of localized visual featuresstored in the memory location.

FIG. 10 is a flowchart of method 1000, in accordance with an exampleembodiment. Method 1000 can be carried out by a UAV, such as the UAV 600illustrated in FIG. 6. In particular, the UAV 600 or a controllerthereof can execute software to carry out method 1000. Additionallyand/or alternatively, the method 1000 can be carried out by a server(e.g., dispatch system), such as local dispatch system 212 or centraldispatch system 210 of FIG. 2.

Method 1000 can begin at block 1002, where the method 1000 involvesoperating an unmanned aerial vehicle (UAV) to begin a flight to adestination, where the UAV uses a primary navigation system to navigateto the destination.

As illustrated at block 1004, the method 1000 can further includeoperating an image capture device coupled to the UAV and having afield-of-view (FOV) of an environment of the UAV to capture a first setof images during the UAV's flight. As described herein, the imagecapture device may be part of a backup navigation system of the UAV.

As illustrated at block 1006, the method 1000 can further includeanalyzing the images to identify visual features within the environment.This step involves the backup navigation system operating in the“feature localization mode” to localize features in the environment.

As illustrated at block 1008, the method 1000 can further includedetermining a current location of the UAV by determining a relationshipbetween one of the identified visual features and a localized visualfeature in a map stored in a memory of the UAV. This step involves thebackup navigation system operating in the “UAV localization mode” todetermine a current location of the UAV. For instance, the backupnavigation system can periodically operate in the UAV localization modein order to determine a location of the UAV. As explained above, thesystem can use a map (e.g., a locally generated or preloaded map) todetermine a current location of the UAV.

As illustrated at block 1010, the method 1000 can further includecomparing the current location to location data from the primarynavigation system. At this step, the system can compare the currentlocation of the UAV as determined by the backup navigation system to thecurrent location of the UAV as determined by the primary navigationsystem.

As illustrated at block 1012, the method 1000 can further includeresponsive to detecting a discrepancy between the current location andthe location data from the primary navigation system, outputting analert indicating failure of the primary navigation system. For example,the system can determine that the discrepancy between the UAV's locationas determined by the backup navigation system and the UAV's location asdetermined by the primary navigation system is greater than a threshold,e.g., a predetermined threshold. If so, the system can determine thatthe primary navigation system may have encountered a failure event ormay have been spoofed and is outputting incorrect data as a result.Thus, responsive to detecting the discrepancy greater than thethreshold, the system can output an alert indicating failure of theprimary navigation system. Additionally and/or alternatively, the backupnavigation system can override to the primary navigation system tonavigate the UAV. As such, the backup navigation system can alsofunction to continually or periodically check the integrity of theprimary navigation system.

VIII. Conclusion

The particular arrangements shown in the Figures should not be viewed aslimiting. It should be understood that other implementations may includemore or less of each element shown in a given Figure. Further, some ofthe illustrated elements may be combined or omitted. Yet further, anexemplary implementation may include elements that are not illustratedin the Figures.

Additionally, while various aspects and implementations have beendisclosed herein, other aspects and implementations will be apparent tothose skilled in the art. The various aspects and implementationsdisclosed herein are for purposes of illustration and are not intendedto be limiting, with the true scope and spirit being indicated by thefollowing claims. Other implementations may be utilized, and otherchanges may be made, without departing from the spirit or scope of thesubject matter presented herein. It will be readily understood that theaspects of the present disclosure, as generally described herein, andillustrated in the figures, can be arranged, substituted, combined,separated, and designed in a wide variety of different configurations,all of which are contemplated herein.

Further, in the context of the present disclosure, data could be treatedto ensure privacy when the data might include (without limitation)information related to: an individual's identity, and individual'slocation, an individual's order history, a UAV's identity, a UAV'slocation, a UAV's flight history, a business's identity, a business'slocation, and/or a business's order history, among others.

Accordingly, in situations in which system(s) collect and/or make use ofinformation about entities (e.g., individuals, UAVs, and/or businesses),data may be treated in one or more ways before it is stored or used, sothat personally identifiable information is removed or otherwise cannotbe discovered by an unauthorized entity/system.

In one example, an entity's identity and/or geographic location may betreated so that no personally identifiable information can be determinedfor the entity. To do so, a system may transmit and/or receive data inthe form of anonymized data streams. That is, data representinginformation related to one or more entities may not provide anyinformation related to respective identities and/or locations of the oneor more entities, thereby maintaining privacy.

In another example, when data is set to include information related toan entity's identity and/or location, the data could be arranged so thatthis information is specified in such a way that only an authorizedentity/system could ascertain a particular identify and/or a particularlocation of an entity. To do so, a system may transmit and/or receivedata in the form of coded data streams in which the information takesthe form a code interpretable only by an authorized entity/system.

In yet another example, data representing information related to anentity's identity and/or location may be encrypted so that only anauthorized entity/system could obtain access to the information. Forinstance, an authorized entity/system could obtain access to theinformation only through use of a previously-obtained security key thatenables access to the information.

In yet another example, data representing information related to anentity's identity and/or location may only be available temporarily. Forinstance, a system could be configured to store such data for a certaintime period and to then permanently delete the data upon detectingexpiration of this time period. Other examples are possible as well.

We claim:
 1. A computer-implemented method comprising: during a flightalong a first flight path to a destination by an unmanned aerial vehicle(UAV), operating an image capture device coupled to the UAV to capture afirst set of images of a field-of-view (FOV) of an environment of theUAV; analyzing the first set of images to identify one or more firstvisual features within the environment; generating a local flight logfor the first flight path, wherein the local flight log associates theidentified first visual features with location information from aprimary navigation system of the UAV; and detecting a failure eventinvolving the primary navigation system of the UAV and responsively:capturing at least one post-failure image using the image capturedevice; identifying, in the at least one post-failure image, one or moresecond visual features within the environment; and detecting a matchbetween at least one of the second visual features and at least one ofthe first visual features, and responsively determining, based at leastin part on (i) location information for the at least one first visualfeature indicated by the local flight log, and (ii) an orientation ofthe UAV corresponding to the capture of the post-failure image, acurrent location of the UAV.
 2. The method of claim 1, wherein the imagecapture device is part of a backup navigation system of the UAV, andwherein the backup navigation system replaces navigational functionalityof the primary navigation system upon detecting the failure event. 3.The method of claim 1, wherein the primary navigation system is reliantupon a global positioning system (GPS).
 4. The method of claim 1,further comprising: responsive to detecting the failure event,determining to fly the UAV to a safe zone; and based on the currentlocation of the UAV, determining at least a portion of a second flightpath to the safe zone.
 5. The method of claim 4, wherein the firstflight path and the second flight path are the same flight path.
 6. Themethod of claim 4, wherein the UAV is deployed from a home base to thedestination, and wherein the safe zone is the home base of the UAV. 7.The method of claim 4, wherein the second flight path has asubstantially similar altitude to the first flight path such that theimage capture device during the second flight path has a substantiallysimilar FOV of the environment to the FOV of the environment that theimage capture device has during the first flight path.
 8. The method ofclaim 1, wherein operating the image capture device to capture the firstset of images along the flight path comprises: using position data fromthe primary navigation system to associate each image with a respectivegeolocation.
 9. The method of claim 1, further comprises localizing theone or more first visual features.
 10. The method of claim 9, whereinlocalizing the one or more first visual features comprises: comparing agiven visual feature identified in a first image from the first imageset to one or more visual features identified in other images from thefirst set; detecting a match between the given visual feature and acorresponding visual feature of at least one other image from the firstset; and determining a location of the given visual feature.
 11. Themethod of claim 10, wherein determining the location of the first visualfeature comprises: using a first geolocation of the first image and asecond geolocation of the second image to determine the location of thefirst visual feature.
 12. The method of claim 10, wherein the algorithmis a triangulation algorithm.
 13. The method of claim 1, furthercomprising: determining a time of the flight and environmentalconditions along the first flight path; selecting a map associated withat least one of the time of the flight and the environmental conditions,wherein the map is representative of at least an area along the firstflight path; and providing the map to the UAV for local storage onmemory coupled to the UAV, before the UAV begins the flight.
 14. Themethod of claim 1, wherein determining the current location of the UAVfurther comprises: matching two or more of the first visual features totwo or more of the second visual features; based on respectivecoordinates indicated by the flight log for the two or more first visualfeatures, determining coordinates of the two or more second visualfeatures; and using the coordinates of the two or more second visualfeatures as a basis for determining the current location of the UAV. 15.The method of claim 1, further comprising, in response to detecting thefailure event, ceasing flight to the destination and initiating flightto a home location.
 16. The method of claim 15, further comprisingnavigating to the home location by, repeatedly updating the currentlocation of the UAV, where updating the current location comprises:capturing at least one additional post-failure image using the imagecapture device; identifying, in the at least one additional post-failureimage, one or more third visual features within the environment; anddetecting a match between at least one of the third visual features andat least one of the first visual features, and responsively determiningan updated current location of the UAV based at least in part on (i)location information for the at least one first visual feature indicatedby the local flight log, and (ii) an orientation of the UAVcorresponding to the capture of the at least one additional post-failureimage.
 17. An unmanned aerial vehicle (UAV) system comprising: a primarynavigation system providing for navigation of the UAV; an image capturedevice configured to capture images of an environment of the UAV and acontroller, wherein the controller is configured to: use the primarynavigation system to begin a flight of the UAV along a first flight pathto a destination; during the flight along the first flight path, operatethe image capture device to capture a first set of images of afield-of-view (FOV) of an environment of the UAV; analyze the first setof images to identify one or more first visual features within theenvironment; generate a local flight log for the first flight path,wherein the local flight log associates the identified first visualfeatures with location information generated by the primary navigationsystem; and detect a failure event involving the primary navigationsystem of the UAV and responsively: (a) capture at least onepost-failure image using the image capture device; (b) identify, in theat least one post-failure image, one or more second visual featureswithin the environment; and (c) detect a match between at least one ofthe second visual features and at least one of the first visualfeatures, and responsively determining, based at least in part on (i)location information for the at least one first visual feature indicatedby the local flight log, and (ii) an orientation of the UAVcorresponding to the capture of the post-failure image, a currentlocation of the UAV.
 18. The UAV of claim 17, wherein the controller isfurther configured to: responsive to detecting the failure event,initiate flight of the UAV to a safe zone; and based on the currentlocation of the UAV, determine at least a portion of a second flightpath to the safe zone.
 19. The UAV of claim 15, wherein the primarynavigation system comprises a global positioning system (GPS) deviceconfigured to receive a satellite signal, and wherein the failure eventis at least one of: (i) a malfunction of the GPS device, and (ii) a lossof the satellite signal.